DOCUMENT RESUME 



ED 288 454 



HE 020 950 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



Yindra, Kathleen J.; And Others 

Medical School Performance as a Predictor of Resident 

Competence. 

Apr 87 

19p.; Paper presented at the Annual Meeting of the 
American Educational Research Association 
{Washington, DC, April 20-24, 1987). 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) 

MFOl/PCOl Plus Postage. 

^Academic Achievement; Competence; Followup Stuc'ies; 
^Graduate Medical Students; Higher Education; 
^Medical Students; Outcomes of Education; 
*Performance; Predictor Variables; *Student 
Evaluation 



ABSTRACT 

The relationships between medical school academic 
achievement and first year residency performance were investigated 
for the 1983 and 1984 graduates of the Medical College of Wisconsin. 
The analyses were designed to control for suspected differential 
resident expectations among the directors of various medical 
specialties^ and were replicated to determine the stability of the 
findings. Completed forms were returned for 64% (119) of the 1983 
graduates and 74% (160) of the 1984 graduates. For both 1983 and 1984 
graduates^ correlations between medical school achievement measures 
and mean residency performance ratings were low, but significant, 
with the exception of 1983 Alpha Omega Alpha membership which was 
also low but insignificant. While the medical school performance of 
outstanding residents was better than that of below average 
residents, it was not possible to distinguish between outstanding and 
average residents, nor between average and below average residents on 
the basis of 'medical school course scores. The findings support the 
use of academic variables in the residency selection process. 
However, their use as a single factor or as a method of screening out 
applicants from further consideration was, at best, justified only in 
the case of extremely high or low medical school achievement 
me a sure s . ( Au t ho r /SW ) 
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This study investigated the relationships between medical school academic 
achievement and first year residency performance for the 1983 and 1984 
graduates of the Medical College of Wisconsin. The analyses were designed to 
control for suspected differential resident expectations among the directors 
of various medical specialties, and were replicated to determine the stability 
of the findings. For both 1983 and 1984 graduates, correlations between 
medical school achievement measures and mean residency performance ratings 
were low, but significant, with the exception of 1983 AOA men±)ership which was 
also low but insignificant. A multiple regression of these data accounted for 
one quarter of the residency rating score variance vAien controlling for rating 
differences between medical specialties. Parallel analyses, between levels of 
residency performance and medical school achievement were carried out using 
analysis of variance. While the medical school performance of outstanding 
residents was better than that of below average residents, it was not possible ' 
to distinguish between outstanding and average residents, nor between average 
and below average residents on the basis of medical school course scores. 
^ This study suj^rts the use of academic variables in the residency selection 

y) process. However, their use as a single factor or as a method of screening 



out applicants from further consideration is, at best, justified only in the 



case of extremely high or low medical school achievement measures. 
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MEDICAL SCHOOL PERPOiWftiCE AS A PREDICTOR OP RESIDENT COMPETENCE 
Kathleen J, YirxJra^ Paul S. Rosenfeld^ M*D. and Michael B. Donnelly^ Ph.D. 
Office of Academic Affairs^ Medical College of Wisconsin 

IWTRODOCnON: In 1973 the ratio of first year residency (PG-1) positions to 
graduating U.S. medical students was 2:1. This ratio declined to 1.2:1 in 
1986 and, in seme competitive specialties^ the ratio fell to 0,5:1 (1), 
Although the resident interview continues to be the most widely used criterion 
in the resident selection process^ the increase in competition for PG-1 
positions has lead residency program directors to place increased reliance on 
acadanic variables (2^ 3) . The use of medical school performance measures for 
selecting residents can be questioned unless their efficacy in predicting 
resident competence can be demonstrated. 

Previous studies have investigated the relationship between resident 
performance and undergraduate CPAs, Medical College Admissions Test scores ^ 
pre-clinical grades and exam scores, clerkship performance ratings, and 
National Board of Medical Examiners performance (4-7) . While only low to 
moderate zero-order correlations have been found, some researchers have 
reported that stronger relationships exist at the highest and lowest 
performance groups (6, 7). Wagoner et al. have indicated that monbership in 
Alpha Onega Alpha (AOA) is of considerable importance in the residency 
selection process (3). Weiss et al. have reported that residents who were 
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PREDICTING RESIDQIT PEREX)SMAtlC:E 

AOA members were more likely to receive above average performance ratings than 
nonHOflonbers (7). Several investigators have studied the utility of the Dean's 
letter of reconmendation in selecting residents and its relationship to post- 
graduate performance. Although disparate findings have been reported (8-11) , 
Stiirroel suggests that a well constructed Dean's letter 

provides a valid indicator of future resident performance (12) . A combination 
of noncognitive and cognitive predictor variables has been shown to increase 
considerably the accxaracy with which residence performance can be predicted 
(4). However y recent evidence suggests that inappropriate conclusions may be 
drawn about the relationship between performance before and after graduation 
if medical specialty is not taken into consideration (6, 12, 13). 

This study was conducted to investigate the relationship between medical 
school academic performance and first year resident performance, as evaluated 
by residency program directors. The analyses were designed to control for 
suspected differential resident expectations among medical specialties and 
were replicated to determine the stability of the findings. 
MET HODCff/Xa f: An evaluation form was mailed to residency directors at the end 
of our 1983 and 1984 graduates' PG-l year. Completed forms were returned for 
64 percent (119) of the 1983 graduates and 74 percent (160) of the graduates 
of the class of 1984. The directors were asked to evaluate the resident on 
six areas of competency: dependability, presentation of historical 
information and physical findings, laboratory and diagnostic tests, problem 
identification, management strategy and overall 
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professional ccxnpetence. The six-point scale ranged from one (outstanding 
performance) to six (unsatisfactory performances) with values of 3 and 4 
demarcating the average range of performance. The six areas of competence 
were factor analyzed using the principal axis method. A one factor solution 
was obtained ( = 30.4% of variance) suggesting the form was measuring the 
perception of residents' overall competence. Thus^ a mean performance score 
was ccxnputed by averaging ratings across the six traits. 

The medical school course scores which wsre analyzed were the average of 
standard scores for courses taught in each of the first three years. Year 1 
average was the mean of the cumulative examination scores in six courses ^ 
year 2 average was the mean of three major second year courses , and year 3 
average was the mean of the three final examinations administered at the end 
of the required clerkships in Medicine, Obstetrics and Gynecology, and 
Psychiatry. Other medical school iTieasures included National Board of Medical 
Examiners Part I and II total scores (NBME I and NBME II), third and fourth 
year medicine clerkship evaluations, election to Alpha Onega Alpha, and a four 
category ranking of the Dean's letter of recommendation. The rank represented 
the writer's assessment of the graduate as a candidate for residency based 
upon cognitive and noncognitive factors. 

In order to determine whether the samples were biased, t-tests were 
conducted on the independent variables. Comparison of the means revealed no 
differences between the graduates for whom residency evaluations were returned 
and those for whom evaluations were not returned. However, for two of the 
nine independent variables, year 2 average and NBME I, the variances of the 
1983 non-returned group were significantly larger than the returned 
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group (p < .05) . Among the 1984 graduates ^ the year 2 average variance was 
also larger for the non returned group (p <.05). 

In order to study the effect of specialty on resident performance ratings, 
the 1984 graduates were assigned to one of six categories based on their PG-1 
specialty; 1) Internal Medicine; 2) Family Practice and Pediatrics; 3) other 
medical specialties (OB/GYN, Psychiatry and Neurology) ; 4) surgical 
specialties (Surgery, Orthopedics, Otolaryngology, Anesthesiology and 
Urology); 5) support specialties; and 6) Transitional. Analysis of variance 
revealed significant differences among specialty groups (F (5, 154) = 4.1, p < 
.001). Residents v^o entered Internal Medicine programs received, 
qualitatively, the lowest ratings (x = 2.7) while those who entered surgical 
specialties received the highest (x = 1.9). The mean rating for Family 
Practice/Pediatrics (x = 2.3), other medical specialties (x =2.2), support 
specialties (x = 2.2), and Transitional (x =2.3) were of equal magnitude. 
Subsequent analyses of variance of medical school achievement scores by 
specialty indicated no significant differences between groups. In fact the 
medicine residents performed as well in medical school as the surgery 
residents. 

This finding suggested that medicine residency directors were more 
stringent raters than directors vAio evaluated residents in surgical 
specialties. Thus, in order to control for residents' medical specialty, the 
nominal-scaled specialty variable was converted to a set of stand-in dummy 
variables. Each category of medical specialty was treated as a separate 
variable, vAiereby, all cases were assigned a score depending upon their 
presence or absence in each of the specialty categories. For statistical 



ERIC 



6 



Page 5 
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reasons only five of the six dunmy variables were included in the subsequent 
analyses. 

Pearson product-mctnent cxtrrelations were computed between residents' mean 
rating score and the medical school variables. Parallel analyses between 
levels of residency performance and medical school performance were carried 
out using one-way analysis of variance. For the analyses of variance four 
independent variable groups were formed based on residents' ratings on the 
overall professional competence trait. It was felt that assigment to 
performance groups based on a discrete variable (overall competence trait) was 
less arbitrary than assignment to groups on the basis of a continuous variable 
(mean rating score) . Residents who received an outstanding rating formed 
group I, those who received a 2 formed the above average group and those given 
a 3 made up the average performance group. Residents who received a rating of 
4, 5 or 6 were ccmbined to form a single low performance group. In order to 
determine the stability of the findings, Pearson product-monent correlations 
and analyses of variance were onployed independently for the class of 1983 and 
1984. 

Multiple regression was onployed in order to evaluate how variables could 
be conbined to find the best prediction equation for residents' mean 
performance. Two sets of predictor variables were entered into the multiple 
regression equation using a hierarchical-stepwise method with pairwise 
deletion of missing data. The set of stand-in specialty variables was entered 
first into the equation. The second set consisted of medical school variables 
vAiich had correlated consistently (for the class of 1983 and 1984) with 
resident performance. Since our data base only included specialty data 
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for the 1984 graduates^ multiple regression analyses were not done for the 
class of 1983. 

RESDLTS; Table 1 presents the zero-order correlations between medical school 
achievement variables and residents' mean performance rating. While some 
correlations were negative^ they were in the expected direction given that a 
rating of 1 represents outstanding performance. Although the correlations 
were low^ all were statistically significant with the exception of AOA^ class 
of 1983. For both the classes of 1983 and 1984, VSEm II had the highest 
correlation (r = -.37) with the mean resident rating. Other significant 
correlations ranged fron r = -.16 (1984 AOA membership) to r = -.33 (1984 NBME 
I). Although the magnitude of the correlations varied across the 1983 and 
1984 samples f a series of t- tests comparing the class of 1983 and 1984 
correlations indicated no statistically significant differences between years. 

In order to obtain a more stable estimate of the relationships between 
medical school and resident measures, correlations were averaged across 
samples using Fisher's r to z transformation, with the exception of AOA 
membership, all of the predictor variables correlated significantly with 
resident performance. However, even the strongest relationship (that between 
NBME II and mean resident ratings) only accounted for 14 percent of the 
variance in resident ratings. 

In order to determine whether the predictive efficiency could be 
increased, multiple regression was employed combining, first, a set of stand- 
in specialty variables and, secondly, the medical school measures. As seen in 
Table 2, controlling for residents' specialty explained 13 percent of the 



ERLC 



8 



Page 7 

PREDICTING RESIDENT EERFOBMAICE 

variance in resident ratings (R = .36) . When the set of medical school 
measures was entered ^ another 13 percent of the remaining variance was 
explained by bJBME Part II scores alone. The three variable weighted model had 
a multiple correlation of R = .51 which accounted for 26 percent of rating 
score variance. When the same procedure was conducted with 1©ME II removed 
frcxn the second set of predictors^ the resulting model incl!:Kaed the year 2 
average score and the fourth year Medicine evaluation. The multiple 
correlation of this model was also R = .51. 

Although the amount of variance explained independently by each predictor 
variable was small ^ the relationships between achievement variables and 
resident performance were stronger at the extremes of the performance 
continua. Analyses of variance of the medical school variables by four levels 
of overall resident performance revealed significant differences between means 
for year 2 average scores^ fourth year medicine clerkship ratings and NBME I 
and II scores for both the class of 1983 and 1984 (see Table 3) . Significant 
differences for the class of 1983 also existed between performance groups for 
year 1 and year 3 average scores, and the third year medicine clerkship 
evaluation. They were not, however, significant for the class of 1984. For 
both samples, there were no significant differences between resident 

performance groups for AOA membership and the Dean's letter rank. 

2 

The omega squared ( ) corresponding to each F-statistic is presented in 

Table 3. While differences were found between performance groups on several 

medical school measures, the amount of achievement variance explained by 

2 

resident performance ( ) was small. Within the resident performance 
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groups, there was considerable variation in achievement for each medical 
school nveasure* For example, while the mean NBME II score was higher for the 
outstanding group than for the low group, some low group residents out 
performed some residents in the outstanding group • 

Post-hoc tests (Tukey B) were conducted on the four variables for which 
there were significant findings in both samples (year 2 average, year 4 
clerkship ratings, NBME I and II). For these variables, residents who 
received outstanding overall performance ratings (group 1) had higher mean 
scores than residents vAio received low ratings (group 4) • The above average 
and average groups (groups 2 and 3) also had higher NBME II means than group 4. 
These findings v^re consistent for the classes of 1983 and 1984* For the class 
of 1984, the group 2 NBME I and year 2 average means were higher than group 
4. Analysis of variance findings indicated that the medical school course 
performance of outstanding residents was consistently higher than that 
achieved by below average residents. However, it is generally not possible to 
distinguish between outstanding residents and average residents, nor between 
average residents and below average residents on the basis of past medical 
school academic criteria* 

DISCOSSION: The present study indicates that there are low, but statistically 
significant relationships betv;een measures of medical school acadenic 
achievement and ratings of residency performance. It should be noted that 
multiple regression analysis of these data produced a prediction equation 
which accounted for only one quarter of the residency rating score variance* 
While this is not large in an absolute sense, it represents the level of 
predictive validity obtained by many educational tests* The low 
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predictive validity may be explained by the assumption ttiat achieve "^nt 
measures predict only students' potential for resident performance and not the 
realization of their potential* 

Wagoner et al. (3) have reported that academic variables are being given 
increasing weight in the residency selection process ^ especially by directors 
of programs in the most competitive specialties. More iniportantly, these 
variables are being used as screening criteria to determine which applicants 
will be interviewed. Because the predictive value of acadfanic measures is 
loWy their use as a sir^le factor or as a screening de\7ice is^ at best, 
justified only in the case of extremely high or low valu3S. 

The lack of a consistently significant correlation between AOA membership 
and residency performance was unexpected. Residency program directors utilize 
AOA mennbership as a significant selection criterion. Vfeiss and colleagues (7) 
found AOA membership to be a valid predictor of the performance of residents 
at the Beth Israel Hospital, Boston. However, acceptance to r^oidencies at 
this hospital is competitive and a high proportion of the residents were 
members of AOA. Our data, does reveal that it was extremely unusual for an AOA 
member to perform poorly as a resident: Or>ly one of the 45 AOA members 
included in this study was ranked in the low performance group. However, most 
of our AOA graduates received above average, rather than outstanding residency 
perfonttance ratings. 

This study demonstrated the confounding effects of specialty on residents' 
ratings. Comparisons of numerical ratings of residents between specialities 
should be approached with caution. The data suggests that 
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specialties differ either in stringency of ratings or in the anphasis given to 
different competencies. 

Finally, it should be noted that throughout this paper the ratings made by 
the program directors were assumed to be valid and reliable. Although this is 
a questionable assumption, these ratings do reflect the : 2rceptions of the 
program directors and as such should not be minimized. 
ODtCUSIOWS MP IMPLICATIONS; These data support the use of acadenic 
variables as one of several factors to be considered in the selecti >n of 
residents, itie use of any of these varia'^les as a sir^le factor or as a 
method of screening out applicants frcro further consideration is not 
justified, and can result in the elimination of many applicants who wi?"" 
perform very veil as residents. 
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table 1 



Pearson Product Moroent Ctorrelatipns Between Average 
PGZ-I Resident Performance Ratings and Madiral School Ashievonent \feriables 
for the 1983 and 1984 Graduates of the Maaical College of Wisconsin 



Average of 

Class of 1983 Class of 1984 1983 & 1984 

(n = 119) (n = 158) Classes ^ 







2 


£ 




2 

£_ 


£ 






£ 


Medical School 




















Year 1 Average 


-.27 


.07 


** 


-.24 


.05 


*** 


-.26 


.07 


ick 


Year 2 Average 


-.28 


.08 


*** 


-.29 


.08 


*** 


-.29 


.08 


*** 


Year 3 Average 


-.26 


.07 


** 


-.25 


.06 


*** 


-.26 


.07 


*** 


Medicine Year 3 


-.28 


.08 


*** 


-.19 


.04 


** 


-.24 


.06 


ick 


Medicine Year 4 


-.27 


.07 


** 


-.23 


.05 


*** 


-.25 


.06 


icic 


AOA Manbership 
Dean's Letter 


-.14 


.02 




-.16 


.03 


* 


-.15 


.02 




.20 


.04 


* 


.22 


.05 


** 


.21 


.04 




Rank 


















N.B.M.E. 




















Part I Ttotal 


-.24 


.05 


** 


-.33 


.11 


*** 


-.29 


.08 


*** 


Part II Total 


-.37 


.14 


*** 


-.37 


.14 


*** 


-.3? 


.14 


-kick 



1 statistical significant value for the average correlations were based on a sample 
size of 119 graduate (Class of 1983) . 

* p< .05 
** p< .01 
*** p< .001 

The negative correlations were in the expected direction given a 6 point resident 
rating scale with 1 = outstanding and 6 = unsatisfactory. 
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St^wise MoLtiple Begcession analyses of 
Significant ^f^dical School Achievement ^iciriables 
in tile Prediction of Average PG-I Resident Ratings 
for 1984 Graduates of the Medical College of Wisconsin 



SET 1 Dummy Variables 

Step 1 Medicine Specialty 
Step 2 Surgical Specialty 



.31 
-.22 



.31 
,35 



.10 
.13 



R change 



.10 
.03 



4.10 
-2.18 



SET 2 



a) Achievement Variables 
Step 3 NBME II 



-.37 



.51 .26 



.13 



-5.15 



b) Achievement Variables (with NHffi II removed) 

Step 3 Year 2 average -.29 .48 .23 .10 -4.48 

Step 4 Medicine 4 rating -.23 .51 .26 .03 -2.35 



1 The first analysis included the dunnmy variables and all significant achievement 
variables. The second analysis included the dvinmy variables and all achievenent 
variables, with nbme II removed. 
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Table 3 



Analysis of Vciriancs of Medical Sctool A:hievanent Measures 
by Four Overall PG-I Professional OGoapetence Groups 
for tiie 1983 anfl 1984 Graduates of the Medical Ctollege of Wisconsin 

Class of 1983 Class of 1984 

v ^ 2 „ « 2 



Medical School 



* p< .05 
** p< .01 
*** p< .001 



Year 1 Average 


3.15 


* 


.054 


2.53 




Year 2 Average 


3.38 


* 


.058 


4.68 


** 


Year 3 Average 


3.59 


* 


.064 


2.41 




Medicine Year 3 


2.91 


* 


.037 


1.53 




Medicine Year 4 


2.98 


* 


.050 


3.56 


* 


Dean's Letter Rank 


1.49 






2.16 




AOA Membership 


1.26 






2.25 




NBME 












Part I Total 


2.83 


* 


.045 


6.13 


*** 


Part II Total 


6.12 


*** 


.119 


7.59 


*** 



.065 
.046 



.091 
.111 
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T&ble 4 




Post-hoc Tests (Tukey B) 
for Year 2 Average^ Medicine 4 Clerkship Rating, and 
National Board Part I & II, Wtiic^ Were Significant in Both 1985 & 1984 Samples 








Resident Performance Groups 

fiiJUvtr rivci.cl^c riVciayc D6XOW AVe." 

Unsatisfact. 




Outstanding 








Above Average 


N.S. 






Average 


N.S. 


N.S. 




Below Average- 
Unsatisfactory 


Yr.2 Ave. (83/84) 
Medicine 4 (83/84) 
NKffi I (83/84) 
NBME II (83/84) 


Yr.2 Ave. ( /84) N.S. 

N.S. N.S. 

NBIffi I ( /84) N.S. 
NBME II (83/84) NBME II (83/84) 




N.S. - Non-significant difference between groups. 
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